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ABSTRACT 

This paper presents a study that: (1) documented the 
course of problems for borderline students in an undergraduate 
medical program; (2) compared documentation by the same tutors of 
weaker students with good students in the same tutorial group; (3) 
documented which of six dimensions (knowledge, problem solving, 
critical appraisal, clinical skills, learning skills, and personal 
qualities) caused these students the most trouble; (4) documented at 
what stage in tJ:e program specific dimensions are particularly 
troublesome; (5) identified what curricular units caused these 
students the most difficulty; and (6) examined whether problem 
students have difficulty with the same dimensions repeatedly, and if 
so, which ones. Findings indicated that problem solving, knowledge, 
and clinical skills were the dimensions with which problem students 
encountered the most difficulty. Newness to the educational system 
and confusion about bases of evaluation also caused students 
problems. Also, the complexity of the patients seen and the broad 
scope of the content caused problen students more difficulties. It 
was found that in most cases the problem students did encounter 
repeated difficulties with the same dimensions. Educational 
recommendations are presented. Contains 20 references. (GLR) 
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An old saw in education is that 10% of the students take up 
90% of the teachers' time. While the percentage of medical 
students barely passing or failing probably smaller than 10% l , 
faculty must direct more of their energies toward them. However, 
what happens to these students or where they encounter difficulties 
is generally not documented in the professional literature. Instead 
most researchers have looked at personal characteristics (2) , 
admissions credentials (3 ' 4) and predicting success in medical school 
<s, e. 7 )# Tnig p aper w in p i 0 t the weaker students' course in a 
problem-based curriculum. Since the evaluation forms in this 
school request a description of behavioral characteristics along 
the same dimensions throughout the program, areas of concern become 
readily apparent to any reader of the record 6,9 . 

An analysis of the pattern of difficulties encountered by weak 
students allows us to test the hypothesis that students have 
trouble with the same dimensions, such as knowledge or clinical 
skills, repeatedly. If this hypothesis is true, and if weaknesses 
are remediable, then it would be to the students' advantage for 
such evaluation data to be available for subsequent faculty 10 . 
Information sharing is consistent with current learning theories, 
that is, providing teachers with background knowledge for proper 
planning on behalf of students". Yet, an AAMC survey indicated 
that 75% of all responding faculty expressed concern about the lack 
of flow of information from one learning unit to another, with this 
being one of the most serious evaluation system problems 12 . 
Objectives . 

1.) document the course of problems for borderline 
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students . 

2. ) compare documentation by the same tutors of weaker 
students with good students in the same tutorial group. Within 
one group is the amount written and its specificity different for 
weaker students compared to of the other students in that same 
group? 

3. ) document which of six dimensions (knowledge, problem 
solving, critical appraisal, clinical skills, learning skills, 
persovial qualities) cause these students the most trouble. 

4. ) document at what stage in the program are specific 
dimensions particularly troublesome. 

5. ) identify what curricular units cause these students the 
most difficulty. 

6. ) document if problem students have difficulty with the same 
dimensions repeatedly, if so, which ones. 

Description of Undergraduate Medical Program. The main forum for 
preclinical learning at this school consists of \ oblem-based 
discussions in groups of approximately six students with a trained 
faculty tutor 13 . The group stays together for a learning unit (>2 
months) and then students are reassigned to the next unit. There 
are five preclinical units, with students continuing to meet in 
small group tutorials during the six clerkship rotations. As 
faculty at this school do not have access to previous student 
evaluations, the same minor problems may be identified repeatedly, 
without improvement"' 6 . Among the various tutor roles, important 
evaluation tasks include collation of all assessment data from that 
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students on the same six dimensions (problem solving, knowledge, 
clinical skills, critical appraisal, learning skills and personal 
qualities) throughout the program. Clinical work is an integral 
part of the entire program. The evaluation form is non-numerical, 
requiring a narrative description of the student's positive and 
negative behaviours in each of the six dimension?. Descriptors for 
each of the six dimensions are listed on the evaluation forms. A 
final grade (satisfactory (S), middle track (M), i.e. may proceed 
to the next unit but requires help in specific areas, or 
unsatisfactory (U) is assigned by the tutor. M or U grades result 
in a formal remedial process. At the present time this school 
does not have a student progress commitwee that reviews all student 
evaluations. Only the student *s advisor receives all evaluations 
for his/her advisee. 

Method 

Data source. For the purposes of this study, students with 
repeated problems were defined as those who received > 1 U's, >_2 
M's or [> 1 U +> 1 M ] . These we call problem students. In six 
recent graduating classes (N =600), 13 problem students were 
identified. Twelve out of 13 of these students were admitted under 
regular admissions criteria, none of them being of disad /antaged 
educational background. These 12 had no identifiable 

characteristics at admissions that would put them at risk. The 
last student had been a medical student in another country and this 
medical school was closed by a repressive government. This 
student, as a refugee, was accepted on humanitarian grounds after 
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a close examination of his previous record. All of the problem 
students eventually graduated. 

For each of these 13 problem students, a register assembled a 
package of photocopied evaluations consisting of that student's 11 
evaluations (from the 5 preclinical units and the 6 clerkship 
rotations), plus those of his/her tutorial peers from each of these 
11 groups. Evaluations from each tutorial group were placed 
together (with student, tutor names and unit identifiers 
eliminated). These packages were numbered sc that the reviewers 
were blinded as to which unit or discipline they were reading, the 
sequence, the student status (problem or peer), and the final grade 
(S, HorU). The tutorial peers were included so that comparisons 
could be made among students in a group and to control for any 
tutor bias. 

Evaluation criteria. A priori, the two reviewers (a clinician 
and an educator both with experience as tutors and unit 
coordinators) developed rating criteria, and an appropriate, 
objective rating form. Criteria for rating the individual concerns 
were based upon the severity and the number of concerns raised. 
These reviewers pretested the criteria and their ability to apply 
them on evaluations not used in the study. 

Review Process. The evaluations for each tutorial group were 
reviewed at the same time. Reviewing the evaluations involved: 
reading the tutor's comments under each of six dimensions, 
documenting those dimensions in which the student was apparently 
having difficulty, if any, rating the level of each concern under 
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each dimension, assessing whether the quality or quantity of 
documentation of each dimension differed from those for the rest of 
the group and assigning a final grade. Quality of statements 
refers to the degree of specificity of statements and whether clear 
examples of behaviours were given. These data were recorded on a 
separate rating form for each of the original tutor's evaluations. 
As a total 630 evaluations were reviewed (13 target students x 11 
units x (4-6) students per group), the 133 evaluations for these 
problem students comprised a subset of a larger study. 

Previous studies 8 '* with this data set revealed that 1.) there 
was good inter-rater reliability among the two reviewers (Kendall's 
Tau = 0.79; and 2.) there was good agreement between the 
independent reviewers' and the original tutors' evaluations (Kappa 
= 0.68,± 0.09). 

Results 

Amount of documentation. The amount of documentation (greater 
than peers, less than peers, no difference) for each of the 6 
general dimensions was compared for all 3 final grade 
(satisfactory, middle track and unsatisfactory) categories. No 
quantitive nor qualitative differences were found in the amount of 
documentation given to the problem students compared to the 
satisfactory ones. 

Comparison of tutors and reviewers on distribution of 
concerns. Our evaluation forms require tutors to check dimensions 
of concern for middle track students only. For their 19 M 
evaluations, the tutors identified 29 areas of concern while the 
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reviewers identified 72 areas of concern for the same evaluations. 
47/72 (65%) of these concerns were of moderate to major severity. 
An example of a moderate concern was recorded on a problem student 
evaluation for clinical skills, " With some difficulties she is 
able to get appropriate history and do simple physical 
examinations. Again, she is well below the average medical 
student. " For the 39 M and U evaluations identified by the 
reviewers a total of 151 concerns, of which 122 (81%) were moderate 
or major concerns were identified. This number cannot be compared 
with the tutors since areas of concern were not labelled for all of 
these students. 

Our previous work 8,9 indicates reviewers identified many more 
concerns than they acted upon in terras of awarding an M or U grade. 
The rest of the data reported here use only the concerns identified 
by the reviewers. These will concentrate on a close look at the 
problem students while giving data about their cohorts for a 
comparative perspective. 

Course of difficulties. Figure 1 shows that these problem 
students had an average of 1.2 concerns per evaluation while their 
cohort had an average of 0.2 concerns per unit. During the 
preclerkship phase, the problem students encountered the most 
concerns in the first and to a lesser extent in the last 
preclinical unit. Most of the concerns raised in units 1 and 5 
were moderate to major. These problem students had fewer separate 
concerns during the middle preclinical units. The pattern for the 
rest of their cohorts is having more difficulties in the first 
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unit and then for the number of concerns to decrease throughout the 
rest of the preclinical units 9 . In the preclerkship most concerns 
are of minor importance compared to moderate and major importance 
(4:2:1). In the clerkship the severity of the concerns becomes 
more evenly distributed among the three levels (3:2:2). However, 
if moderate and major concerns are collapsed, then there are 4:3 
moderate and major concerns compared to concerns of minor 
importance. Table 1 gives individual student data in terms of the 
number of times each one had difficulty with each dimension and 
other data about these students. Table 2 summarizes the dimensions 
of difficulty in terms of preclerkship and clerkship performance 
and gives the cohort comparisons. 

Students take clerkship rotations in different order thus we 
can look at them in calendar order compared to specific 
disciplines to see whether it was timing in clerkship or discipline 
which caused the most problems. When the rotations were looked at 
in calendar order, there does not appear to be any pattern of 
problem evolution. It is the disciplines themselves which appear 
to cause problems. The Medicine clerkship caused the students the 
most problems with surgery causing the second most problems. 

Dimensions of concern. The pattern of difficulties is similar 
for problem students and their cohorts. For the cohorts, problem 
solving and clinical skills causes the most difficulties throughout 
with knowledge also causing many concerns in the clerkship. As 
Figure 2 shows for these problem students problem solving, 
knowledge, and clinical skills cause the most difficulties 
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throughout. In terms of moderate and major concerns, the problem 
students encounter most difficulties in problem solving and 
knowledge. Table 3 shows the number of concerns per dimension per 
unit. 

Repetition of same concerns. Table 4 shows the repetition of 
the same concerns by dimension with the same students. Among these 
13 problems students, the same dimensions were identified as 
problems twice with the same student 20 separate times. All but 2 
students had trouble with the same dimensions at least twice in the 
program, with 10 of them having the same problems at least three 
times. 

For the student who had trouble with problem solving seven 
times and knowledge four times his evaluations consistently raised 
the same types of concerns within each dimension. For example, on 
the last preclinical unit, this student's problem solving 
evaluation rtiad, "... consistently had difficulty in dealing with 
the initial aspects of problem-solving. He could not identify 
issues and set them in some sort of priority. As a result, his 
discussions of a problem were either completely unfocussed or took 
an inordinately long time to become clarified". This same 
student's evaluation for problem solving on a clerkship was, "able 
to obtain the necessary details at history and physical 
examination, but unable to synthesize and therefore, unable to 
prioritize problems. Cannot problem solve at clinical level." The 
comments for knowledge for this student show consistency also. For 
a preclinical unit, the tutor wrote, " already had a fairly wide 



knowledge base entering into this unit and expanded it to some 
extent. However, there war repeated concern expressed that his 
knowledge was superficial and that he had difficulty tying together 
knowledge from different areas to deal with a specific problem. 
This concern remained at unit end". In a clinical unit, the 
knowledge evaluation for this student read, "has probably a 
satisfactory knowledge base but the knowledge is fragmented and has 
difficulty applying the information". 

One student had trouble with the same five out of six 
dimensions four times out of 11 units. The only dimension in which 
she did not encounter repeated difficulties was personal qualities. 
Another student encountered difficulties with problem solving, 
clinical skills and personal qualities in 5 out of the 11 units. 
A third student encountered difficulty with five out of six 
dimensions at least twice. The dimension in which he did not 
encounter difficulties was critical appraisal. 

Having trouble with one dimension tends to lead to concerns 
with other dimensions. As other concerns are identified such as in 
knowledge or problem solving, concerns in personal qualities tend 
to escalate. All but one problem student had at least one 
difficulty with personal qualities during medical school. There are 
very few concerns with personal qualities in the cohort after Unit 
1. Concerns with knowledge were also correlated with concerns in 
problem so ving and to a lesser extent learning skills. For 
example, with one student, knowledge deficiencies were identified 
as causing difficulties in problem solving in Units 1, 4 and 
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surgery. The preclerkship tutors both stated that they thought 
that this student's lack of skills in problem solving would 
probably not be a concern once her knowledge base became adequate. 
In Unit 1 the tutor wrote for problem solving, 11 She has difficulty 
summarizing and integrating knowledge around a problem, thus she 
does not demonstrate good problem solving skills" . Her tutor in 
Univ 4 wrote, "her poor knowledge base or her inability to link 
knowledge with hypotheses leads to very poor hypothesis 
generation". This same student had repeated difficulties with 
learning skills, as stated by her tutors, due to her inability to 
assess her own knowledge base and her own performance. Her tutor 
for surgery stated, "She thinks -hat she can identify deficiencies 
herself, but I have no evidence that this is so. However, when 
deficiencies are pointed out to her she responds appropriately. 
(Learning skills in this program generally means self -directed 
learning. ) 

Discussion 

There seems to be specific points in the program when most 
concerns emerge. As expected and also found with their cohorts, in 
the first unit many concerns were raised. So much of what the 
students are being evaluated on, and the whole system of learning 
is new for them. During this unit the students are first exposed 
to the demands of small group, problem-based learning including the 
acquisition of material on their own. Others have documented that 
it takes some students awhile to get used to this type of learning 
14. The objectives of this unit include an emphasis on three 
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perspectives of biology, including an understanding of basic 
pathophysiology concepts such as inflammation and infection; 
population « a. luding concepts of epidemiology and health 
prevention; behavior including concepts of deve]opment and 

motivation. For many students it may be the first time they are 
thinking about some of these concepts. During the middle three 
units of the pre-clerkship phase, the problem students' and their 
cohorts' records indicate fewer concerns. It seems that the 
students learn what is expected of them in this preclinical, or 
classroom phase of their learning. These are also the organ system 
units in which the greatest emphasis is placed on the acquisition 
of bi >?.ogical knowledge. Then during the last preclinical unit, an 
integrative block, the number of areas of concern escalate again. 
This unit may be difficult for these somewhat weaker students as 
the format of learning changes here, because the students have to 
access patients on their own to form the basis for their tutorial 
discussions. There is a greater emphasis on the three perspectives 
again. 

As with their cohorts, during the clerkships, more 
difficulties are identified. Most of the concerns are raised in 
medicine and surgery. Medicine and surgery are very intense 
experiences with supervisors working closely with the students. 
These clerkships allow repeated opportunities for 1-1 student 
supervision. The complexity of the patients seen and ne broad 
scope of the content may also cause these pr^jlem students more 
difficulties. In both medicine and surgery many concerns were 
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moderate and major. In the cohort sample the most concerns were 
raised in family medicine and psychiatry. 

Problem solving, knowledge and clinical skills are the 
dimensions in which the problem students encounter the most 
difficulty. These dimensions can be remediated. Students lacking 
a good knowledge base can acquire content independently through 
studying materials in texts or other library resources or through 
faculty contact. Improving one's knowledge base may also enhance 
problem solving skills since they are likely content specif ic iS ' 16 
Students can also be assisted in approaches to problem solving, 
such as gathering data in a systematic way, asking questions which 
test specific hypotheses, avoiding early closure, or learning to be 
more organized. Clinical skills can be learned and improved 
provided the students obtain proper feedback on their clinical 
work. This feedback should be specific, relate to behaviors and 
constructive. Students can also learn clinical skills through 
modelling. 

Repeated difficulties with potentially remediatable dimensions 
gives support for why previous evaluation information should be 
made available to faculty 10. Perhaps if the students had been more 
forthcoming in discussing their own shortcomings, or if the faculty 
had known about them, the students could have been given more 
remedial guidance. Perhaps if this school had had a student 
progress committee whose function is to review all student 
evaluations and to look longitudinally at the students who 
encounter repeated problems, we may also have remediated some of 
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these problems earlier. We believe that early identification and 
remediation can lead to overcoming difficulties. 

Explanations for the phenomena of the escalation of personal 
qualities in association with other difficulties include that the 
students get defensive abou^ their problems, they use denial or try 
to cover up their shortcomings and they work at ways to avoid 
detection of their problems. 

Educational significance. The overwhelming conclusion from 
this study is that in most cases the problem students did encounter 
repeated difficulties with the same dimensions. This finding leads 
to four significant educational recommendations. These have been 
discussed in the literature, but without much change in practice. 
This study further supports the need to implement these 
recommendations. 1) Students must be evaluated on several 
dimensions other than knowledge during the preclerkship phase of 
their training 17 ' 18 ) . 2) This multifaceted evaluation needs to be 
on-going with an emphasis on formative as well as summative 
evaluation 18 ~ 20 . 3) Difficulties must be documented early; faculty 
and students must be prepared to remediate problems once they are 
identified «' 10 ' 1 »' 1 ». 4) Previous evaluation information should be 
made available to faculty in a judicious manner 8,10 . If problem 
students had been more forthcoming in discussing their own 
shortcomings with subsequent faculty or with advisers, they might 
have been offered more remedial guidance and might have avoided 
some of their repeated problems. 
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TABLE 1 

Concerns and progress of 13 Problem Students 

Number of times each student encountered difficulty with each dimension 



Problem Students (letter Id's) 



Dimension 


a 


b I 


c 


d|l 


e II 


f I 


9 I 


h 


i I 


i I 


k 


I 


m 


Problem Solving 


3 


4 


2 


8 


2 


2 


3 


2 


1 


2 


0 


5 


5 


Knowledge 


1 


4 


3 


4 


2 


3 


1 


1 


1 


2 




3 


5 


Critical Appraisal 


5 


4 


2 


2 


1 


0 


0 


0 


1 


1 




2 


0 


Clinical Skills 


1 


4 


4 


2 


2 


1 


0 


3 


0 


2 




7 


4 


Learning Skills 


0 


4 


3 


? 


2 


1 


0 


2 


0 


1 




2 


I 2 


Personal Qualities 


1 


? 


3 


2 


3 


1 


1 


1 


0 


2 




5 


I 5 


[Totals 


II 11 


II 22 


II 17 


II 20 


II 12 


II 8 


I 5 


9 


I 3 II 10 


I 5 


II 24 


I 21 



Othflr data these students ==X 




LaJI 


L_bJ 


LcJ 




i e I 




LiU 


L_L) 


i I 


k I 1 


1 I 1 


m 


Leave of Absence 


n 


n 


n 


n 


n 


n 


n 


y~j 


n 


~T~| 


n 


n 


n 


Remedial work 


n 


y 


y 


y 


n 


n 


y 


n 


n 


y 


n 


n 


y_ 


Graduation delay 
yes = # months 


n 


6 


12 


18 


18 


n 


12 


n 


n 


18 


n 


n 


n 


Pass LMCC 


y 


n 


y 


y 


y 


y 


n 


n 


y 


y 


y 


y 


y 


first try 
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TABLE 2 



Concerns and Progress of 13 Problem Students 

Number of times students encountered difficulty with each dimension and comparison 
to the cohort sample 



1 

1 

i 


Problem Studentts 

!: ti m 

i 




1 

i ' 




Cohort (n= 


597) :; 

1 


i 

1 


.. J _ll 

j! Pre Clerk || 


Clerk J 


Total 


,.JI.Pre 


Clerk 


1 lj Clerk j! 


Total 


S Problem Solving 


L i . ..is j 


;:.23.::]!.: 


39 




5. 


:j : \:;is: ii 


20 


j Knowledge 


":r:.9.:.T 


~ 22 t: 


31 




7 


:"jr...:i2::;s 


19 -■;! 


j Critical Appraisal 


I 14 ]| 


: 5 .j. 


"19 


. - . 


4 ; 


"li 2 . ;| 




I Clinical Skills 


I 15 | 


16 | 


31 


II 


3 


II 18 li 


31 i 


I Learning Skills 


' I a"'. I 


12 || 


20 


"1 


4" 


"I"' 7 || 


11 ■ ! ; 


| Personal Qualities 


...1...: 12 i 


15 || 


27 


zi . 


;io .... 




21 1 




□lzz 


zzzjl: 








□i:zz:zji 


;.j 


[Grand total 


□izi^zi 




167 


.jl*:. 


43" 


es ii 


108 | 



Average Number of Concerns 
Per Problem Student 



Perclerkship 14,8 
Clerkship 15.5 
Overall 15,18 



Average Number of Concerns! 
Per Cohort Student jj 

0.014 'j 
0.018 i 



TABUS 3 



Number of concerns/dimension/unit 





PS 


K 


CA 


CS 


LS 


PQ 


Unitl 


6 


6 


4 


8 


5 


6 


Unit 2 


3 


5 


4 


5 


3 


3 


Unit 3 


4 


1 


2 


7 


3 


2 


Unit 4 ' 


6 


6 


2 


3 


3 


4 


Unit 5 


4 


3 


5 


5 


3 


7 


Med 


6 


8 


3 


7 


6 


5 


FM 


6 


3 


4 


4 


3 


6 


Sur 


6 


7 


2 


4 


4 


4 


Ped 


3 


2 


0 


2 


0 


5 


Psy 


6 


6 


3 


3 


6 


5 


O-G 


5 


4 


2 


2 


3 


2 



TABLE 4 



REPETITION OF SAME CONCERN BY DIMENSION WITH SAME 
STUDENT 



problem know- critical clinical learning personal 
solving ledge appraisal skills skills qualities 



number 
of times 
same 
concern 
was 

identified 



2x 


5 


2 


2 


3 


5 


3 


3x 


2 


3 


0 


1 


1 


2 


4x 


1 


2 


1 


3 


1 


0 


5x 


2 


1 


1 


0 


0 


2 


6x 


0 


0 


0 


0 


0 


0 


7x 


0 


0 


0 


1 


0 


0 


8x 


1 


0 


0 


0 


0 


0 
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FIGURE 1 
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FIGURE 2 



Number of times problem students 
encounter difficulty with 6 dimensions 
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